Method of searching data to identify images of an object captured by a camera system

ABSTRACT

A camera system comprises an image capturing device, object detection module, object tracking module, and match classifier. The object detection module receives image data and detects objects appearing in one or more of the images. The object tracking module temporally associates instances of detected objects, each of which has a signature representing features of the detected object. The match classifier matches object instances by analyzing data derived from the signatures. The match classifier determines whether the signatures match.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/068,433, filed Mar. 11, 2016, which is a continuation of U.S. patentapplication Ser. No. 14/183,201, filed Feb. 18, 2014, now U.S. Pat. No.9,317,753, which is a division of U.S. patent application Ser. No.13/543,668, filed Jul. 6, 2012, now U.S. Pat. No. 8,655,020, which is adivision of U.S. patent application Ser. No. 12/397,276, filed Mar. 3,2009, now U.S. Pat. No. 8,224,029, which claims benefit of U.S.Provisional Patent Application No. 61/033,349, filed Mar. 3, 2008, andU.S. Provisional Patent Application No. 61/033,284, filed Mar. 3, 2008,the last two of which are incorporated herein by reference in theirentireties. The application is also related by subject matter to U.S.patent application Ser. No. 14/183,154, filed Feb. 18, 2014, now U.S.Pat. No. 9,076,042, which is a division of U.S. patent application Ser.No. 13/543,668, filed Jul. 6, 2012, now U.S. Pat. No. 8,655,020.

TECHNICAL FIELD

This disclosure relates generally, but not exclusively to, videosurveillance and, more particularly, to matching objects represented inmultiple images.

BACKGROUND

Automated security and surveillance systems typically employ videocameras or other image capturing devices or sensors to collect imagedata. In the simplest systems, images represented by the image data aredisplayed for contemporaneous screening by security personnel and/orrecorded for later reference after a security breach. In those systems,the task of detecting objects of interest is performed by a humanobserver. A significant advance occurs when the system itself is able toperform object detection and tracking, either partly or completely.

In a typical surveillance system, for example, one may be interested intracking a detected object such as, for example, a human being, avehicle, or an animal, that moves through the environment. Existingsystems capable of tracking detected objects attempt to track objectsusing motion prediction and tracking of selected features in consecutiveframes of video. Other techniques, such as the SIFT method, attempt toprecisely represent the appearance of an instance of an object such thatthe representation can be used to match multiple instances of an objectirrespective of their temporal proximity. Known tracking systems,however, suffer from one or more of (1) inability to be trained, (2)lack of integration with object searching, indexing, and classificationsystems, (3) inadequate object tracking and search performance, and (4)ineffective cross camera tracking capabilities.

SUMMARY BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial diagram of a camera system according to oneembodiment.

FIG. 2 is a simplified block diagram of one of the image capturingdevices in the system shown in FIG. 1.

FIG. 3 is a block diagram depicting of a method of generating an inputfor the match classifier of FIG. 2, according to one embodiment.

FIG. 4 is a block diagram of the match classifier of FIG. 2.

FIG. 5A is a block diagram showing one of the initial N−1 steps in thematch classifier shown in FIG. 4.

FIG. 5B is a depiction of acceptance and rejection thresholds utilizedin the step shown in FIG. 5A.

FIG. 6 is a flow chart depicting a method of utilizing the matchclassifier, according to one embodiment.

FIG. 7 is a flow diagram of a match classifier training method.

FIG. 8 is a flow chart of a match classifier training method.

FIG. 9 is a flow chart of another match classifier training method.

FIG. 10 is a block diagram of an object tracking system.

FIG. 11 is a flow chart of an object tracking method.

FIG. 12 is a hybrid block diagram and flow chart of another objecttracking system.

FIGS. 13A and 13B are hybrid block diagrams and flow charts of aninter-camera tracking method.

FIG. 14 is a block diagram of an object indexing system.

FIG. 15 is a pictorial diagram of an object search method.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

With reference to the above-listed drawings, this section describesparticular embodiments and their detailed construction and operation.The embodiments described herein are set forth by way of illustrationonly and not limitation. Those skilled in the art will recognize inlight of the teachings herein that there is a range of equivalents tothe example embodiments described herein. Most notably, otherembodiments are possible, variations can be made to the embodimentsdescribed herein, and there may be equivalents to the components, parts,or steps that make up the described embodiments.

As one skilled in the art will appreciate in light of this disclosure,certain embodiments are capable of achieving certain advantages over theknown prior art, including some or all of the following: (1) unifyingobject tracking, indexing, and searching methods, (2) implementing atrainable match classifier for object tracking, indexing, and searching,(3) ability to search for instances of an object captured prior to arule violation, (4) ability to recognize previously captured objectsafter a new object type is added to a detection library, (5) ability toadapt object tracking, indexing, and searching to environmental changes,(6) ability to add new features to optimize detection and classificationaccuracy, (7) implementing a high-speed tracking system that allowsaccurate tracking even with relatively low frame rates, (8) implementingan accurate tracking system on an embedded platform using inexpensivehardware, (9) ability to improve object detection accuracy andclassification accuracy by feeding back accurate tracking data, and (10)ability to automatically track objects across cameras in real time.These and other advantages of various embodiments will be apparent uponreading the remainder of this section.

For the sake of clarity and conciseness, certain aspects of componentsor steps of certain embodiments are presented without undue detail wheresuch detail would be apparent to those skilled in the art in light ofthe teachings herein and/or where such detail would obfuscate anunderstanding of more pertinent aspects of the embodiments.

A camera system comprises an image capturing device, an object detectionmodule connected to the image capturing device, an object trackingmodule connected to the object detection module, and a match classifierconnected to the object tracking module. The image capturing device hasa field of view and produces image data representing multiple images ofthe field of view. The object detection module receives the image dataand is operable to detect objects appearing in one or more of themultiple images. The object tracking module is operable to temporallyassociate instances of a first object detected in a first group of themultiple images. The first object has a first signature representingfeatures of the first object derived from the images of the first group.The match classifier is operable to match object instances by analyzingdata derived from the first signature of the first object and a secondsignature of a second object. The second object is detected in a secondimage distinct from the images of the first group. The second signaturerepresents features of the second object derived from the second image.The match classifier analyzes the first set of data to determine whetherthe second signature matches the first signature. The match classifieroperates to match object instances in a manner that has been determinedat least in part by a training process that automatically configures thematch classifier using a set of possible object features.

A method tracks an object captured by a camera system. The methodcaptures first and second images of a field of view of the camerasystem, detects a first object captured in the first image and a secondobject captured in the second image, and produces a first signature ofthe first object and a second signature of the second object. The firstsignature represents a first set of features of the first object and thesecond signature represents a second set of features of the secondobject. The method produces data derived from the first and secondsignatures. The data corresponds to a first subset of features selectedfrom the first set and a second subset of features selected from thesecond set. A training process automatically determines the features ofthe first and second sets that are to be selected for inclusion in thefirst and second subsets. The method analyzes the data to determinewhether the second object is an instance of the first object.

A method generates index elements of objects in images captured by acamera system. The method captures a first group of multiple images ofan object in a field of view of the camera system, detects the object inthe first group of multiple images, and generates a first set ofsingle-frame signatures of the object. The single-frame signatures ofthe first set correspond to features of the object derived from themultiple images of the first group. The method produces a first averagesignature of the object. The first average signature is an average ofthe single-frame signatures of the first set. The first averagesignature is a first index element of the object. The method produces acumulative signature of the object. The cumulative signature is aweighted average of the single-frame signatures of the first set. Themethod captures a second image of the object, detects the object in thesecond image, and generates a second single-frame signature derived fromthe second image. The method updates the cumulative signature byweighing the second single-frame signature and averaging the weightedsecond single-frame signature with the cumulative signature. The methoddetermines whether to accept or reject the updated cumulative signatureand the first average signature as a match. The first average signatureis updated based on the second single-frame signature when the updatedcumulative signature and the first average signature are accepted as amatch. A second average signature of the object is produced based on thesecond single-frame signature when the updated cumulative signature andthe first average signature are rejected as a match. The second averagesignature is a second index element of the object.

A method searches data to identify images of an object captured by acamera system. The method captures a first object in an image of a fieldof view of the camera system. The first object has a first signaturerepresenting features of the first object. The method displays an imageof the first object on a display, accepts user input indicative of aselection of the first object, and makes determinations as to whetherthe first signature matches a group of signatures corresponding to a setof objects captured in images of the field of view of the camera system.The method sends a notification when a second signature of a secondobject of the set matches the first signature.

A camera system comprises an image capturing device, an object detectionmodule connected to the image capturing device, a user interfaceconnected to the image capturing device, a match classifier, an objecttracking module connected to the match classifier and the objectdetection module, and a search module connected to the match classifierand the user interface. The image capturing device has a field of viewand produces image data representing multiple images of the field ofview. The object detection module receives the image data and isoperable to detect objects appearing in one or more of the multipleimages. The user interface includes a display for displaying objectsdetected by the object detection module and an input device operable toselect objects detected by the object detection module in response touser commands. The match classifier is operable to match objectsdetected by the object detection module. The object tracking module isoperable to temporally associate instances of objects detected by theobject detection module and is also operable to call on the matchclassifier to determine whether to associate instances of objects. Thesearch module is operable to retrieve stored images of objects selectedby the input device and is also operable to call on the match classifierto locate a stored image of a selected object.

Camera System

FIG. 1 is a pictorial diagram of a camera system 100 according to oneembodiment. The camera system 100 includes image capturing devices 102,a user interface 104, and a remote storage/processing unit 106 (e.g., aremote server) connected to each other in a network 108. The network 108may include any type of wired or wireless network. Although the camerasystem 100 of FIG. 1 includes multiple image capturing devices 102connected in a network, the camera system 100 may include a single imagecapturing device 102. The image capturing devices 102 may include aninternal storage system 110, comprising a hard drive (HD) 111, and ametadata database (DB) 112. For example, the image capturing devices 102may include a storage system described in commonly owned U.S. patentapplication Ser. No. 12/105,971 and U.S. Pat. No. 8,427,552, entitled“Content Aware Storage of Video Data” and “Extending the OperationalLifetime of a Hard-Disk Drive Used in Video Data Storage Applications,”respectively, both of which are incorporated herein by reference intheir entireties. The user interface 104 includes a display 114 and aninput device 116. Image capturing devices 102 capture images of theirrespective fields of view and generate image data representing theimages. It is to be understood that images may refer to still images ormotion video images. The image data is communicated over the network 108to the user interface 104, and images of one more of the fields of vieware presented on the display 114. The input device 116 is operable toallow a user to provide user feedback information for the camera system100. Image data may also be communicated over the network 108 to theremote storage/processing unit 106.

FIG. 2 is a simplified block diagram of one of the image capturingdevices 102. The image capturing device 102 may be a high-resolutionvideo camera such as a megapixel video camera. The image capturingdevice 102 may also capture data from outside the visible spectrum(e.g., thermal energy). In addition to the storage system 110, the imagecapturing device 102 includes an image processing unit that includes avideo analytics module 200 for analyzing images captured by the imagecapturing device 102. The image processing unit need not be containedwithin a housing 202 of the image capturing device 102 as depicted inFIG. 2. Moreover, the remote storage/processing unit 106 may alsoinclude an image processing unit.

The video analytics module 200 includes a number of modules forperforming various tasks. For example, the video analytics module 200includes an object detection module 204 for detecting objects appearingin the field of view of the image capturing device 102. The objectdetection module 204 may employ any known object detection method suchas motion detection or blob detection. The object detection module 204may include the systems and use the detection methods described incommonly owned U.S. Pat. No. 7,627,171, entitled “Methods and Systemsfor Detecting Objects of Interest in Spatio-Temporal Signals,” theentire contents of which are incorporated herein by reference.

The video analytics module 200 also includes an object tracking module206 connected to the object detection module 204. As used herein, theterm “connected” means logically or physically connected directly orindirectly through one or more intermediaries. The object trackingmodule 206 is operable to temporally associate instances of an objectdetected by the object detection module 204. The object tracking module206 generates metadata corresponding to objects it tracks. The metadatamay correspond to signatures of the object representing the objects'appearance or other features. The metadata may be transmitted to themetadata database 112 for storage. The object tracking module 206 isdescribed in more detail below.

The video analytics module 200 also includes a temporal objectclassification module 208. The temporal object classification module 208is operable to classify an object according to its type (e.g., human,vehicle, animal) by considering the object's appearance over time. Inother words, the object tracking module 206 tracks an object formultiple frames (i.e., multiple images), and the temporal objectclassification module 208 determines the object's type based upon itsappearance in the multiple frames. The temporal object classificationmodule 208 may combine information regarding the trajectory of an object(e.g., whether the trajectory is smooth or chaotic, whether the objectis moving or motionless) and the confidence of classifications made byan object classification module 210 averaged over multiple frames. Forexample, classification confidence values determined by the objectclassification module 210 may be adjusted based on the smoothness oftrajectory of the object. The temporal object classification module 208may assign an object to an unknown class until the object is classifiedby the object classification module a sufficient number of times and apredetermined number of statistics have been gathered. In classifying anobject, the temporal object classification module 208 may also take intoaccount how long the object has been in the field of view. The temporalobject classification module 208 may make a final determination aboutthe class of an object based on the information described above. Thetemporal object classification module 208 may also use a hysteresisapproach for changing the class of an object. For example, a thresholdmay be set for transitioning the classification of an object fromunknown to a human, and that threshold may be larger than a thresholdfor the opposite transition (i.e., from a human to unknown). Thetemporal object classification module 208 may generate metadata relatedto the class of an object, and the metadata may be stored in themetadata database 112.

The video analytics module 200 also includes the object classificationmodule 210 connected to the object detection module 204. In contrast tothe temporal object classification module 208, the object classificationmodule 210 determines an object's type based upon a single instance(i.e., single image) of the object. The object classification module mayinclude the systems and use the methods described in commonly owned U.S.patent application Ser. No. 12/397,287, entitled “Dynamic ObjectClassification,” the entire contents of which are incorporated herein byreference. The temporal object classification module 208 may aggregatethe classifications made by the object classification module 210.

The video analytics module 200 also includes an object indexing module212 connected to the storage system 110. The object indexing module 212is operable to generate signatures for objects. The signatures may bestored in the metadata database 112 and may act as index elements forvideo images of the objects. The object indexing module 212 is describedin more detail below.

The video analytics module 200 also includes an object search module 214connected to the storage system 110 and the user interface 104. Theobject search module 214 is operable to search through signatures storedin the storage system 110 to identify whether an object was present inpreviously captured images. The object search module 214 is described inmore detail below.

The video analytics module 200 also includes an object categorizationmodule 216 connected to the object classification module 210. The objectcategorization module 216 is operable to prioritize tracking whenmultiple objects are detected in an image. For example, the objectcategorization module 216 is operable to use classification informationgenerated by the object classification module 210 to establish an orderfor attempting to associate objects of a current frame with objectsdetected in past frames. The object categorization module 216 isdescribed in more detail below.

Data generated by the video analytics module 200 may be used by a rulesengine 220 to determine whether one or more user-specified rules havebeen violated. For example, the rules engine 220 may trigger an alarmthat is presented on the display 114 of the user interface if a human isdetected in the field of view of one of the image capturing devices 102.

As used herein, the term “module” is a component that may comprise oneor more hardware circuits or devices and/or one or more softwareroutines, functions, object or the like. A module may also be entirelyhardware, entirely software, comprise firmware, or comprise somecombination of the foregoing. As used herein, the term “system” refersto a tangible thing.

Match Classifier

The video analytics module 200 also includes a match classifier 218connected to the object tracking module 206, the object indexing module212, and the object search module 214. The match classifier 218 isoperable to receive an input pattern z representing signatures of twoobjects and determine whether the signatures match (e.g., whether thesignatures are sufficiently similar). The match classifier 218 may beused by the object tracking module 206, the object indexing module 212,and the object search module 214 to assist the modules with theirvarious operations. Thus, a unified framework for tracking, indexing,and searching may be achieved via the match classifier 218.

The match classifier 218 is operable to match objects based upon thesignatures (e.g., features or appearance characteristics) of theobjects. For example, the match classifier 218 receives data (i.e., theinput pattern z) related to the signatures of two objects captured indifferent frames and determines whether the objects correspond to thesame physical object.

A signature of an object may be made of a subset of features {circumflexover (F)}={f.sub.k1, f.sub.k2, . . . , f.sub.km} selected from a set offeatures F={f.sub.1, f.sub.2, . . . , f.sub.n}. The elements of{circumflex over (F)} may be viewed as some transformation of an imageregion R of an object. Thus, a signature x of an object may take on thefollowing form:x=(f^1=fk1(R)f^2=fk2(R)f^m=fkm(R))  (1)##EQU00001##

The features {circumflex over (f)}.sub.1, {circumflex over (f)}.sub.2, .. . , {circumflex over (f)}.sub.m of an object may correspond to anumber of appearance characteristics such as, but not limited to, aspectratio, hue, intensity, edge orientations, texture, corner features, rawimage pixel data, normalized saturation, and localized descriptors suchas Scale-invariant feature transform (SIFT) features. The featuresinclude both color features (e.g., hue and saturation) and gray-scalefeatures (e.g., intensity and edge orientation). This allows the matchclassifier 218 to work with both color and black and white images. Animage of an object may be divided into regions or subwindows, and thefeatures {circumflex over (f)}.sub.1, {circumflex over (f)}.sub.2, . . ., {circumflex over (f)}.sub.m may correspond to the appearancecharacteristics of the object in the regions or subwindows. Moreover,the features {circumflex over (f)}.sub.1, {circumflex over (f)}.sub.2, .. . , {circumflex over (f)}.sub.m may represent feature vectors (e.g.,histograms in which the histogram bins correspond to vector components)of the appearance characteristics and may be used by the matchclassifier 218 to determine whether objects match. For example,histograms of the edge orientations of an object may be constructed fordifferent regions (e.g., subwindows) of the object's image. In otherwords, an image of an object may be divided into subwindows, and edgeorientations may be calculated for each pixel of the subwindows. Theedge orientation of a pixel may be derived using a steerable filter(e.g., using a Gaussian derivative filter in multiple directions). Forexample, an image may be convolved with a first derivative of a Gaussiandistribution oriented at zero and 90 degrees, which may be used as basisfilters to synthesize numerous orientations. Gaussian derivativeresponses may be computed at eight orientations for each pixel, and foreach pixel, the orientation with the maximum response may be chosen asthe direction for the pixel. Using a steerable filter allows dominantdirections to be assigned to the pixels of a subwindow, and allows ahistogram of the directions to be constructed for the subwindow. Forexample, for a given pixel, a steerable filter may be used in multipledirections to generate multiple responses, and the directioncorresponding to the maximum directional derivative response is assignedas the direction of the pixel. A signature x of an object may capturethe uniqueness of the object while allowing for appearance variationsamong different instances of the object. Although not required, asignature of an object may be computed with maximum resolution availablefor the camera system 100 and may not be affected by post-factodown-sampling or compression.

FIG. 3 is a block diagram showing a method 300 that may be used togenerate an input z for the match classifier 218 based on a signature xfrom a first object and a signature y from a second object. Generally,the signatures x and y from the first and second object are combined toform the input z (step 302). For example, feature {circumflex over(f)}.sub.1 of signature x may be combined with feature {circumflex over(f)}.sub.1 of signature y. The signatures x and y may be combinedmathematically using combination functions such as a histogramintersection or a concatenation of features. A combination function maybe thought of as a distance measure d between the features. Manydifferent distance measures may be used to come up with the input z suchas an L.sub.1 distance, a Manhattan distance, an L.sub.2 distance, and aBhattacharyya distance. Thus, many combinations or distance measures arepossible for feature {circumflex over (f)}.sub.1 of the signature x andfeature {circumflex over (f)}.sub.1 of signature y, for example. Thedistance measures may correspond to a measure of similarity between thefeatures; instances of the same object may be “close” in feature space(e.g., small distance measure), while instances of different objects maybe “far away” in feature space (e.g., large distance measure). Whencombining edge orientation histograms from the signatures x and y, forexample, concatenation of the edge orientation histograms may serve as adistance measure d. The input z is supplied to the match classifier 218,and the match classifier 218 makes a decision as to whether the firstand second object match. As described below, a training process mayautomatically choose the best features of the signatures x and y,together with the best combination or distance measure, to achieve ahigh level of accuracy in matching objects. The output of the matchclassifier 218 may correspond to a decision step value s(z) as describedbelow. The decision step value s(z) may indicate whether the first andsecond object match, and may include a value corresponding to aconfidence level in its decision.

The match classifier 218 may be represented by a classification problemdefined generally by a classifier function .GAMMA.(z), in which twoobjects represented by the input pattern z are declared a match when.GAMMA.(z)>0 or as a non-match when .GAMMA.(z)<0. Generally theclassifier function .GAMMA.(z) is parameterized with a set ofparameters, and the input z is composed of a combination of the featuresdescribed above. The magnitude of the output of the classifier functionmay reflect the level of confidence of the match classifier 218 in itsdecision. This level of confidence may constitute a measure ofsimilarity (or dissimilarity).

An embodiment of the structure of the match classifier 218 will now bedescribed in more detail with reference to FIGS. 4-6. The matchclassifier 218 includes multiple steps 400 (N decision steps) configuredin a cascade configuration (i.e., step 1, followed by step 2, followedby step 3, . . . , followed by step N) as shown in FIG. 4. The N stepsoperate to determine whether two signatures match. Unlike a knowncascade classification system proposed in Paul Viola & Michael Jones,“Robust Real-Time Face Detection,” International Journal of ComputerVision, pp. 137-154 (2004), each of the initial N−1 steps of the presentembodiment is operable to make one of three decisions: (1) accept twoobjects as a match, (2) reject the objects as a match (i.e., declare anon-match), and (3) forward the decision onto the next step. Forexample, the input z is supplied to step 1, and step 1 decides whetherto (1) accept the objects as a match, (2) reject the objects as a match,or (3) forward the decision to step 2. The decision to accept, reject orforward is based on a value produced by the step (i.e., the decisionstep value s(z)). The final or Nth step is operable to either (1) acceptthe objects as a match or (2) reject the objects as match.

FIG. 5A is a block diagram showing one of the initial N−1 steps 400 inmore detail. Each step 400 includes one or more stages 500. For each ofthe stages 500, a combination (e.g., a distance measure orconcatenation) of one of the features {circumflex over (f)}.sub.1,{circumflex over (f)}.sub.2, . . . , {circumflex over (f)}.sub.m from afirst object and a corresponding one of the features {circumflex over(f)}.sub.1, {circumflex over (f)}.sub.2, . . . , {circumflex over(f)}.sub.m from a second object is extracted (represented by blocks 502)from the input z and supplied to its corresponding stage 500. Forexample, a combination of the feature {circumflex over (f)}.sub.1 fromthe first object and the feature {circumflex over (f)}.sub.1 from thesecond object may be supplied to the first stage 500. Each stage 500,therefore, has a corresponding feature combination associated with it.The stage/feature combination may be predetermined by a learningalgorithm during a training process as described below. Moreover, afeature combination supplied to the first stage (stage 1) of FIG. 5A,for example, may be different from or the same as the featurecombination supplied to the second stage (stage 2).

In general, a stage 500 may be represented by a stage function g whichis defined as follows:g:(x,y).fwdarw..gamma.g.epsilon.G(x,y).epsilon.z and.gamma..epsilon.[−1,1]  (2)in which G represents a general set from which the specific stagefunction g is chosen and (x, y) represents a combination of a featureextracted from signature x and a feature extracted from signature y. Asdescribed above, a combination of features supplied to a stage 500 maybe considered a distance measure d. The set G may, therefore, be writtenas G=D.times.T where D is a set of all possible distance measurescalculated at step 302 of FIG. 3 (e.g., d.epsilon.D,d: (x,y).fwdarw.z)and T represents a set of possible transformations (i.e., mappings) suchthat for t.epsilon.T, t: z.fwdarw..gamma.. Thus, the stage function gmay take on the following expanded form:g.sub.i({circumflex over (f)}.sub.k.sup.x,{circumflex over(f)}.sub.k.sup.y)=t(d({circumflex over (f)}.sub.k.sup.x,{circumflex over(f)}.sub.k.sup.y))  (3)

A stage 500 represents a discriminant function that includes a weightvector {right arrow over (w)} and an activation function .beta.. Asnoted above, the features {circumflex over (f)}.sub.1, {circumflex over(f)}.sub.2, . . . , {circumflex over (f)}.sub.m may represent featurevectors. Additionally, a distance measure d between feature vectors mayrepresent a distance vector {right arrow over (d)}. The stage functionmay be rewritten as follows:g.sub.i({right arrow over (d)}.sub.k)=.beta..sub.i({right arrow over(w)}.sub.i{right arrow over (d)}.sub.k)  (4)where {right arrow over (w)}.sub.i{right arrow over (d)}.sub.krepresents the inner product of the weight vector {right arrow over (w)}and the distance vector {right arrow over (d)}.sub.k. The activationfunction .beta..sub.i may be any function such as, but not limited to, asigmoid function or a radial basis function. The activation function.beta..sub.i is used to map the inner product of the weight vector{right arrow over (w)} and the distance vector {right arrow over(d)}.sub.k to a value between zero and one. The scalar value .gamma. maybe determined by calculating a differential of the discriminantfunction. Unlike known cascade classifiers that include the samediscriminant function for all stages, the discriminant functions for thestages of the present embodiment may be different from each other.Moreover, the weight vector {right arrow over (w)}.sub.i and activationfunction .beta..sub.i for each stage 500 are automatically determinedduring training as described below.

Each step 400 is a linear combination (represented by scale and summingblock 504) of one or more of the stages 500. In other words, the scalarvalues .gamma. of the stages 500 are scaled and summed to produce thedecision step value s(z). In general, a step function may be representedmathematically as follows:s(z)=.SIGMA..sub..A-inverted..sub.i.alpha..sub.ig.sub.i(x.sub.k,y.sub.k)where .SIGMA..sub..A-inverted..sub.i.alpha..sub.i=1  (5)

Because s(z) (i.e., the decision step value) is a convex combination ofstages, s(z) and g(x,y) have the same range. Weighting coefficients.alpha. are chosen by a learning algorithm during training. The decisionstep value is compared (represented by block 506) to one or both of anacceptance threshold .tau..sub.a and a rejection threshold .tau..sub.rto determine whether two objects match, to reject the objects as amatch, or to forward the decision to the next step 400. The comparisonmay be represented as follows:Accept if .tau..sub.a<s(z).ltoreq.1Forward if .tau..sub.r<s(z).ltoreq..tau..sub.aReject if −1<s(z).ltoreq..SIGMA..sub.r  (6)

FIG. 5B illustrates one example of the acceptance and rejectionthresholds in the interval [−1, +1]. The acceptance threshold.tau..sub.a and the rejection threshold .tau..sub.r are chosen duringtraining by a learning algorithm based on user-specific false positiveand false negative rates. Each step 400 may have values for .tau..sub.aand .tau..sub.r that are different from or the same as the values theother steps 400 have. A decision to “Accept” implies that the classifieris confident that two objects match. A decision to “Forward” impliesthat the classifier is unsure and defers the decision to the next step.A decision to “Reject” implies that the classifier is confident that twoobjects do not match. At any step, if the decision is not to “Forward”to the next step 400, the accept/reject decision is made at that pointand the evaluation is complete. The decision step value s(z) at thatstep is the output value for the match classifier 218. The decision stepvalue may correspond to a decision confidence level for thecorresponding step 400. For example, a decision step value close to onemay represent that the corresponding step 400 is more confident that twoobjects match compared to a decision step value slightly above theacceptance threshold .tau..sub.a. Alternatively, an increase in thedecision step value may not necessarily correspond to a higher decisionconfidence level (i.e., a probability that the decision was correct).The confidence level associated with each decision step value may beempirically estimated during training as objects are correctly andincorrectly matched with different decision step values. The confidencelevels of the decision step values are described in more detail below.As noted above, the final stage (stage N) in the match classifier 218 isforced to always accept or reject:Accept if 0<s(z)ltoreq.1Reject if −1<s(z).ltoreq.0  (7)

FIG. 6 is flow chart depicting an operational flow 600 of the matchclassifier 218 by way of example. First, the signatures of a first andsecond object are combined (step 602). The first and second objects maycorrespond to the same physical object. Step 602 correlates with step302 of FIG. 3. The input z is transmitted to the first step 400 of thematch classifier 218 (step 604). The combined features (e.g., thedistance measures) that correspond to the first through S.sup.th stages500 are identified in and chosen from the input z (step 606).Alternatively, rather than transmitting the input z to the first step400, the combined features used by the stages 500 of the first step 400may be selected and only those combined features may be transmitted tothe first step 400. The combined features are supplied to theirrespective stages 500, and the stages 500 map the combined features toscalar values .gamma. (step 608). The scalar values are scaled (i.e.,weighted) and summed to produce a decision step value s(z) (step 610).The decision step value is compared to one or more of the acceptancethreshold .tau..sub.a and the rejection threshold .tau..sub.r (step612). If the decision step value is greater than the acceptancethreshold .tau..sub.a, the first and second objects are accepted as amatch (step 614). If the decision step value is less than or equal tothe rejection threshold .tau..sub.r, the first and second objects arerejected as a match (step 616). If the decision step value is greaterthan the rejection threshold .tau..sub.r but less than or equal to theacceptance threshold .tau..sub.a, the input z is forwarded to the secondstep 400 (or, in the alternative, only those feature combinations usedby the second step 400 are transmitted to the second step 400) (step618). The first and second objects may be accepted or rejected as amatch at any step 400 within the cascade.

Training the Match Classifier

A method of training the match classifier 218 will now be described.Conventional classifiers may be trained with the learning algorithmAdaBoost or some variant of AdaBoost. While AdaBoost has proved itsworth in some applications, the algorithm and objective function used inthat learning process have some limitations. For example, for AdaBoostto be effective, distance measures of matching and non-matching objectscannot overlap each other significantly in feature space. In otherwords, the features of the objects should separate a classificationspace well. Moreover, because AdaBoost uses weak learners, a largecollection of weak learners may be necessary to form a full classifiercapable of achieving a desired accuracy.

According to one embodiment, an alternative objective function andlearning algorithm called Sequential Discriminant Error Minimization(SDEM) is used to train the match classifier 218. SDEM was proposed inSaptharishi, “Sequential Discriminant Error Minimization: The Theory andits Application to Real-Time Video Object Recognition,” (Carnegie MellonUniversity, 2005), the entire contents of which are incorporated hereinby reference. SDEM can deal with features or distance measures that donot necessarily separate a classification space well. Unlike AdaBoostand other similar boosting techniques, SDEM may use weak learners orother discriminant functions that are not necessarily considered weak.Thus, the number of steps 400 and stages 500 of an object classifier maybe significantly less using SDEM for training compared to AdaBoost. Fora given feature, SDEM is often capable of learning the best classifierfor the corresponding feature space. The best features of objects,distance measures, and transformations may be automatically selected fora given classification problem.

Generally, the SDEM algorithm is used to train the combination of stages500 for each step 400. As defined in equation (3), a stage 500 includesa stage function g.sub.i(x,y) that is equal to a transformation t of adistance measure d of features {circumflex over (f)}.sub.k.sup.x and{circumflex over (f)}.sub.k.sup.y (i.e., t(d({circumflex over(f)}.sub.k.sup.x, {circumflex over (f)}.sub.k.sup.y))). The trainingtask chooses the best transformation t, the best distance measure d, andthe best feature {circumflex over (f)}.sub.k, such that when aparticular stage 500 is added to a step 400, the performance of theobject classifier is maximized. In other words, the SDEM algorithmselects the transformation t, distance measure d, and the feature{circumflex over (f)}.sub.k for a particular stage so as to maximize anobjective function. The set F of features and the set of distancemeasures D may be finite, and the set T of transformations may becontinuous and differentiable. Following the SDEM algorithm, for eachchoice of a discrete pair ({right arrow over (f)}.sub.k, d.sub.j), asearch is performed in the set T to identify the transformation t thatperforms best on a training data set. The search in the set T may beperformed using standard unconstrained optimization techniques such as,but not limited to, a Quasi-Newton optimization method. Once the besttransformation t is identified for each choice of ({right arrow over(f)}.sub.k, d.sub.j), the best feature/distance measure pair may beselected according to an estimated generalization error {circumflex over(.epsilon.)}(g). Selection of the best feature/distance measure pair maybe written as follows:g({circumflex over (f)}.sub.k.sup.x,{circumflex over(f)}.sub.k.sup.y)=min.sub..A-inverted.(k,j),.A-inverted.t.epsilon.T{{circ-umflexover (.epsilon.)}(t(d.sub.j({circumflex over(f)}.sub.k.sup.x,{circumflex over (f)}.sub.k.sup.y)))}  (8)

When each stage 500 is added to a step 400, there may be n uniquefeature types and b unique distance measures. Thus, a total of n.times.bpairs ({circumflex over (f)}.sub.k, d.sub.j) may be explored when addinga new stage 500. One of the properties of the SDEM algorithm is thatwhen a stage 500 is added to a step 400, the addition of the stage 500improves the performance of the object classifier on the training dataset. If a new stage cannot be identified that improves the performanceof the object classifier, the SDEM algorithm automatically terminates.Alternatively, rather than waiting for SDEM algorithm to automaticallyterminate, a number of stages 500 of a step 400 may be determined by auser. In other words, the SDEM algorithm terminates training when amaximum number of stages 500 set by the user is reached or when no stage500 can be added that will improve performance.

The SDEM algorithm selects a series of features, distance measures, andtransformations such that when combined, the series outperforms a singleone of the features/distance measures/transformations. For example,while the aspect ratio of an object might be a poor feature to use formatching, when combined with local gradient information, the aspectratio may improve the matching accuracy corresponding to the localgradient information alone. A number of simple features, distancemeasures, and transformations may be combined to create a highlyaccurate match classifier 218. The training task creates super-featuresby combining a set of appearance features and distance measures for twoobjects.

The training task for building the stages 500 of first step 400 will nowbe described in more detail with reference to FIGS. 7 and 8. Thefollowing description is also applicable to the stages 500 of the othersteps 400. FIG. 7 is a block diagram of a method 700 of generatingtraining data. A sequence of images of objects may be presented to auser for verification as to whether the images correspond to the sameobject or different objects. For example, two images may besimultaneously presented, or sequentially presented, to the user forverification. When the images correspond to the same object, a matchexample is created (step 702). When the images correspond to differentobjects, a non-match example is created (step 704). Distance measuresd.sub.1, d.sub.2, . . . , d.sub.m are used for the features of the matchand non-match examples. The features {circumflex over (f)}.sub.1,{circumflex over (f)}.sub.2, . . . , {circumflex over (f)}.sub.k of theobjects of the match and non-match examples may have more than onedistance measure associated with it. For example, b different distancemeasures may be available for one or more of the features {circumflexover (f)}.sub.1, {circumflex over (f)}.sub.2, . . . , {circumflex over(f)}.sub.k. Thus, d.sub.1 and d.sub.2 may both correspond to feature{circumflex over (f)}.sub.1 of the objects. The features {circumflexover (f)}.sub.1, {circumflex over (f)}.sub.2, , {circumflex over(f)}.sub.k, of the match and non-match examples are combined by usingone or more of the different distance measures associated with theparticular features. For example, if d.sub.1 corresponds to {circumflexover (f)}.sub.1, d.sub.1 is used to combine the feature {circumflex over(f)}.sub.1 of one object of a match example with the feature {circumflexover (f)}.sub.1 of the other object of the match example. The featuresf.sub.1 of the objects of the other match and non-match examples mayalso be combined. After a number of match and non-match examples arecreated and the features {circumflex over (f)}.sub.1, {circumflex over(f)}.sub.2, . . , {circumflex over (f)}.sub.k of the objects of theexamples are combined using the appropriate distance measures d.sub.1,d.sub.2, . , d.sub.m, the examples and combinations may be used in atraining algorithm to train the match classifier 218 (step 706).

FIG. 8 is a flow chart of a training method 800 that may be used totrain the match classifier 218. The combinations of features {circumflexover (f)}.sub.1, {circumflex over (f)}.sub.2, . . . , {circumflex over(f)}.sub.m of the objects of the match and non-match examples (i.e.,feature combinations 802) are identified and associated with thedistance measures (e.g., d.sub.1, d.sub.2) that were used to combine thefeatures (steps 804 a, 804 b, 804 c). For example, for the distancemeasure d.sub.1, the combinations of the appropriate feature {circumflexover (f)} formed by using d.sub.1 are identified (step 804 a).

After the feature combinations 802 corresponding to the distancemeasures d.sub.1, d.sub.2, . . . , d.sub.m are identified, the besttransformation t is selected for each of the distance measures d.sub.1,d.sub.2, . . . , d.sub.m (steps 806 a, 806 b, and 806 c). Because aparticular feature {circumflex over (f)} is associated with eachdistance measure, the best transformation is selected not only for thedistance measure, but also for the feature associated with the distancemeasure. The transformations may be selected based on standardoptimization techniques. A transformation t may be viewed as a decisionboundary that separates the feature combinations 802 of the match andnon-match examples. Thus, the best transformation t corresponds to adecision boundary that best separates the match and non-match examplesfor the corresponding feature {circumflex over (f)} and distancemeasure. In the context of the discriminant function made up of theweight vector {right arrow over (w)}.sub.i and the activation function.beta..sub.i, selection of the best transformation t corresponds toselection of the activation function .beta..sub.i and the components ofthe weight vector {right arrow over (w)}.sub.i that best separate thematch and non-match examples. The activation function .beta..sub.i maybe selected from a set of multiple function types such as, but notlimited to, a sigmoid function and a radial basis function (e.g., aGaussian function). Thus, unlike known cascade classifiers, a firststage may include a different type of discriminant function compared toa second stage.

After the transformations for the distance measures d.sub.1, d.sub.2, .. . , d.sub.m are selected, a value of an objective functioncorresponding to each distance measure/transformation combination iscalculated (steps 808 a, 808 b, and 808 c). The objective function maybe proportional to a measure of errors (e.g., declaring a match anon-match or vice versa) or it may be a nonlinear but monotonicallyvarying function of the classification errors. The calculated values ofthe objective function may be related to the number and/or severity ofclassification errors made by the different distancemeasure/transformation combinations. For example, a first calculatedvalue may be related to the number of classification errors made by thedistance measure d.sub.1 and its corresponding transformation. Thecalculated values of the objective function are compared, and thedistance measure/transformation combination that has the maximumcalculated value is selected for the first stage 500 of the first step400 (step 810).

After the distance measure (and its associated feature) andtransformation are selected for the first stage 500, the featurecombinations 802 of the different match and non-match examples areweighted with different weights taking into account the decisions madeby the first stage 500 (step 812). Each of the feature combinations 802of the match and non-match examples may be thought of as correspondingto data points in function space. The feature combinations 802 of thematch and non-match examples may be weighed as a function of how closetheir corresponding data points are to the decision boundary representedby the first stage 500. For example, the feature combinations 802corresponding to data points close the decision boundary in featurespace may be weighed with higher weights compared to featurecombinations 802 with data points far away from the decision boundary sothat a second stage 500 may be trained by focusing more on those featurecombinations 802 that somewhat confused the first stage 500. Thedistance between a data point and the decision boundary of the firststage 500 may be related to the scalar value .gamma. calculated for theexample corresponding to the data point.

After the feature combinations 802 of the examples are weighted, thebest transformations are again selected for the distance measuresd.sub.1, d.sub.2, . . . , d.sub.m (steps 806 a, 806 b, and 806 c arerepeated). The feature combinations 802 corresponding to the distancemeasures d.sub.1, d.sub.2, . . . , d.sub.m however, are now weighted,and the best transformation t for each of the distance measures d.sub.1,d.sub.2, . . . , d.sub.m is selected taking into account the first stage500. The best transformation t may correspond to the transformation thatresults in the greatest increase of the objective function value. Thevalues of the objective function are again calculated and compared todetermine the distance measure/transformation for the second stage 500(steps 808 a, 808 b, 808 c, and 810 are repeated). To build a thirdstage 500, the feature combinations 802 of the match and non-matchexamples are re-weighted, in which the examples that were somewhatconfusing to the first and second stages 500 are given higher weights.Again, steps 806 a, 806 b, and 806 c are repeated, but featurecombinations 802 corresponding to the distance measures d.sub.1,d.sub.2, . . . , d.sub.m are now re-weighted, and the besttransformation t for each of the distance measures d.sub.1, d.sub.2, . .. , d.sub.m is selected taking into account the first and second stages500. The values of the objective function are again calculated andcompared to determine the distance measure/transformation for the thirdstage 500 (steps 808 a, 808 b, 808 c, and 810 are repeated). The processof choosing the best distance measure, transformation, and weighting thefeature combinations 802 of the examples through each iteration oftraining a new stage may be thought of as a gradient ascent in functionspace, or as a process to increase the total value of the objectivefunction.

Once the first step 400 is trained, the thresholds .tau..sub.a and.tau..sub.r are selected such that desired false positive and falsenegative rates may be attained. Moreover, as the stages 500 areconstructed for the first step 400, the weighting coefficients .alpha.are also selected. For example, as each stage 500 is added to the firststep 400, the weighting coefficients .alpha. for the stages are adjustedto find the values for the weighting coefficients .alpha. thatcorrespond to the lowest overall error rate for the first step 400. Forexample, the weighting coefficients .alpha. may be selected by using aline search optimization strategy.

After the first step 400 is trained, the stages 500 of a second step 400may be trained. However, fewer than all of the match and non-matchexamples used to train the first step 400 may be used to train thesecond step 400. For example, only those match and non-match examplesthat the first step 400 could neither accept nor reject may be used totrain the second step 400. In other words, the second step 400 istrained with the examples that had a corresponding decision step valuegreater than the rejection threshold .tau..sub.r, but less than or equalto the acceptance threshold .tau..sub.a. This allows the second step 400to focus only on those examples that the first step 400 found confusing.

Once the match classifier 218 is trained with the examples, the matchclassifier 218 may continue through other training steps to refine thedistance measures/transformations selected for the different stages 500.A high-level approach to training the match classifier 218 is shown inthe flow chart of FIG. 9, which illustrates a training method 900. Imagedata 901 (e.g., raw video data) is supplied to a simple base or seedsystem that is capable of basic detection, tracking, and classificationof objects. The base system detects, tracks, and classifies objectsrepresented in the image data 901 and generates metadata correspondingto the objects (step 902). The base system selects a set of objects thatare detected and tracked (step 904). The selection of the objects maydepend on the amount of time an object was in the field of view of animage capturing device 102, or may depend on how confident the basesystem was in its classification of an object. Other rules may bespecified for dictating whether an object is selected by the basesystem.

Images of the objects selected by the base system are presented on adisplay to a user so the user can manually label the objects as match ornon-match examples. The user manually labels the objects, and thelabeled examples are supplied to the match classifier 218 being trained(step 906). Feature combinations may be calculated for the labeledexamples using the distance measures d.sub.1, d.sub.2, . . . , d.sub.m.The feature combinations of the manually labeled examples may correspondto the feature combinations 802 described above. The match classifier218 is trained with the feature combinations 802 such as according tothe training method 800 as described above with reference to FIG. 8(step 800). Image data 901 are supplied to the match classifier 218. Thematch classifier 218 determines whether objects represented in the imagedata 901 match and generates metadata representing matched andnon-matched objects (step 910). Each match and non-match generated bythe match classifier 218 has a match confidence level associated withit. The match confidence level corresponds to the decision step value ofthe step 400 that declared the match or non-match. The match confidencelevels generated by the match classifier 218 are analyzed to identifyinstances of objects that were confusing to the match classifier 218(e.g., matches and non-matches with a low match confidence levels). Theperformance of the match classifier 218 is evaluated to determinewhether the match classifier's performance is acceptable (step 912).

To determine whether the match classifier's performance is acceptable, adisjoint test set may be used in which matches and non-matches of thedisjoint test set are known prior to classification by the matchclassifier 218. The image data 901 supplied to the match classifier 218may correspond to the disjoint test set, and the classifications made bythe trained object classifier may be compared to the actual matches andnon-matches. From this comparison, the performance of the matchclassifier 218 may be determined. If the performance is not equal to orabove some predefined performance level, the confusing objects arepresented to the user for manual labels (step 904). The user labels theconfusing objects and the new labeled matches and non-matches are usedto retrain the match classifier 218 (steps 906 and 908). When the matchclassifier 218 is retrained, the distance measure/transformationcombinations for the different stages 500 may be updated based on thenew labeled examples. The retrained match classifier is used to classifyobjects represented in the image data 901 and the performance of theretrained match classifier is evaluated (steps 910 and 912). Theretraining process may continue until the performance of the matchclassifier 218 is acceptable. When the performance of the matchclassifier 218 is acceptable, it may be deployed (step 914). Thetraining process may be recast into the following steps: 1. Manuallylabel some small fraction of a dataset. 2. Train a match classifier 218using the fraction of the dataset. 3. Use a newly trained classifier toautomatically label the complete dataset. 4. Select a set ofautomatically labeled data points that were confusing to the matchclassifier 218. 5. Manually label the confusing data points. 6. Repeatthe training with all the new labeled data points. 7. Go to step 3.

Given the steps—of stages architecture of the match classifier 218, anew step may be added to the match classifier 218. This new step may betrained to correct mistakes made by the match classifier 218.Alternatively, a new step may be trained to replace the last or N.sup.thstep of the match classifier 218. After the match classifier 218 istrained by the process described above, the match classifier 218 mayperform various operations such as tracking, generating indexingelements, and performing object-based (e.g., appearance-based) searches.In other words, the match classifier 218 need not be trained separatelyfor each of its various operations.

The manner in which the match classifier 218 operates to match objectinstances has been determined at least in part by a training orself-learning process that automatically configures the match classifier218 using a set of possible object features. That configuration mayinclude one or more of the following determinations: (1) selecting whichfeatures from the set of possible features to use; (2) determining theorder in which the selected features are evaluated; (3) determining howmultiple features are combined (i.e., the weighting coefficients used tocombine multiple variables); (4) selecting distance measures from a setof available distance measures (e.g., L.sub.2, Manhattan,Bhattacharyya). Other aspects of the match classifier 218 may beconfigured during the training process. The training process may takeplace off-line prior to deployment of the match classifier 218 and/oron-line during on-line operation of the match classifier 218.

Match Confidence

The decision step value s(z) is correlated with the match classifier'sestimate as to how similar it thinks two objects are (e.g., matchconfidence). The correlation may not be linear, i.e., the step 400 maygenerate a high positive value, but the objects may not match.Typically, by virtue of the training process, the higher the value ofs(z), the less likely that the step 400 made a mistake. The confidencelevel associated with a certain value of s(z) may be calculated by firstdefining an indicator function .epsilon.(.GAMMA.(z)) in which:(.GAMMA.(z)=s(z))={0, Instances match and s(z)>00, Instances do notmatch and s(z).ltoreq.01, Instances match and s(z).ltoreq. 01, Instancesdo not match and s (z)>0   (9) ##EQU00002##

A confidence function, .psi.(.GAMMA.(z)), may be defined as theprobability that the step 400 declared two instances to be a match, andthat it was correct for an output of s(z)=.nu.. Thus, for a smallquantization interval [.nu.−.DELTA., .nu.+.DELTA.], the confidencefunction may be expressed as follows:.psi.(.GAMMA.(z))=P.sub..epsilon.,.OMEGA.|.GAMMA.(.epsilon.(.GAMMA.(z))=−0,.omega.=match|s(z)<.nu.+.DELTA.)−P.sub..epsilon.,.OMEGA.|.GAMMA.(.epsilon.(GAMMA.(−z))=0,.omega.=match|s(z)<.nu.−.DELTA.)  (10)

Note, it may be considered that the step 400 declares two instances tobe a match when s(z)>0, i.e.,P.sub.,.OMEGA.|.GAMMA.(.omega.=match|s(z)>0)=1. Thus, for .nu.>0,equation (10) can be expressed as:.psi.(.GAMMA.(z))=P.sub..epsilon.|.OMEGA.,.GAMMA.(.epsilon.(.GAMMA.(z))=−0|.omega.=match,0<s(z).ltoreq..nu.+.DELTA.)−P.sub..epsilon.|.OMEGA.,.GAMMA.(epsilon.(.GAMMA.(z))=0|.omega.=match,0<s(z)<.nu.−.DELTA.)  (11)Equation (11) represents the true positive rate when.nu..epsilon.[.DELTA., 1−.DELTA.] and s(z).epsilon.[.nu.−.DELTA.,.nu.+.DELTA.].

Similarly, the confidence of a step 400 in declaring that the objectbelongs to the negative class for .nu..ltoreq.−.DELTA. may be expressedas:.psi.(.GAMMA.(z))=P.sub..epsilon.|.OMEGA.,.GAMMA.(.epsilon.(.GAMMA.(z))=−0|.omega.=match,.nu.+.DELTA..ltoreq.s(z).ltoreq.0)−P.sub..epsilon.|.OMEGA.−,.GAMMA.(epsilon.(.GAMMA.(z))=0|.omega.=match,.nu.−.DELTA.<s(z).ltoreq−.0)  (12)

Equation (12) represents the true negative rate when.nu..epsilon.[−1+.DELTA., −.DELTA.] and s(z).epsilon.[.nu.−.DELTA.,.nu.+.DELTA.]. Thus, if the probability (as defined in equations (11)and (12)) that the step 400 is correct for any observed output values(z)=.nu. is high, then the step 400 is considered to be confident inits answer. For this self assessment of confidence, a probabilitymeasure {circumflex over (p)}.sub..epsilon.|.OMEGA.,.GAMMA. is estimatedfrom the training examples and the confidence function .psi.(.GAMMA.(z))is inferred. If the confidence function .psi.(.GAMMA.(z)) is less than acritical confidence threshold .psi..sub.c for a given output value fors(z), then the step 400 is considered to be diffident or confused forthat output value. Diffident classifications are forwarded to the nextstep 400. Thus, the confidence function .psi.(.GAMMA.(z)) may be usedduring training to identify the objects that are confusing to an matchclassifier 218. If discriminant functions that constitute the stages 500are a good approximation to the Bayes optimal decision boundary, thenthe decision step value s(z) will be monotonically related to theconfidence function .psi.(.GAMMA.(z)). For the initial steps 400, theobject classifier may not approximate the Bayes decision boundary well.Thus, the decision step value s(z) for a given step 400 and theconfidence function .psi.(.GAMMA.(z)) may not always be monotonicallyrelated.

The confidence function .psi.(.GAMMA.(z)) may be used to determine theacceptance threshold .tau..sub.a and the rejection threshold .tau..sub.rfor the different steps 400. As opposed to other cascade classifierarchitectures, if the confidence function .psi.(.GAMMA.(z)) for matchesis monotonically increasing with the decision step value s(z), theacceptance threshold .tau..sub.a may be chosen such that true positiveconstraints are met. If the confidence function .psi.(.GAMMA.(z)) doesnot increase monotonically with the decision step value s(z), then theacceptance threshold .tau..sub.a may remain saturated at 1, i.e., noinstances are accepted as matches in the corresponding step 400. Thelack of monotonicity indicates that in the match region, the decisionboundary does not reflect the Bayes optimal classifier sufficientlywell. Similarly, the rejection threshold .tau..sub.r is chosen if thenon-match confidence is monotonically related to the decision step values(z). In practice, the non-match class may be far more densely populatedin feature space than the match class. Thus, while the monotonicrelationship may not exist for the match class in early steps, it mostlikely exists for the non-match class in the early steps. The matchconfidence level corresponding to the match classifier 218 is theclassification confidence associated with the decision step value s(z)generated by the step 400 that either accepted or rejected a match.

Tracking

Tracking may be thought of as locating an object in each video frame orimage, and establishing correspondences between moving objects acrossframes. Tracking may be performed within a single image capturing device102 or across multiple image capturing devices 102. In general, theobject tracking module 206 may use object motion between frames as a cueto tracking, while also relying on the match classifier 218 fortracking. Using the match classifier 218 for tracking may enableaccurate tracking even when some occlusion is present, motion issomewhat erratic, and a fair number of crisscrossing objects are presentin the field of view of an image capturing device 102.

FIG. 10 is a block diagram of a system 1000 used by the object trackingmodule 206 for tracking objects. The object tracking module 206 includesa motion modeling module 1002 that is operable to estimate changes oflocation of an object between sequential images or frames of a field ofview of an image capturing device 102. The motion modeling module 1002may predict the location and size of an object in a new frame based uponits previously estimated trajectory and velocity. The motion modelingmodule 1002 may be used to rank observations of objects in a currentframe by their likelihood or probability of corresponding to aparticular object that is being tracked. The object tracking module 206may generate a list of tracked objects or targets that have beendetected in previous frames. For example, the list of tracked objectsmay include ten objects, the current frame may include five detectedobjects, and a ranking (e.g., one through five with one being thehighest ranking) of the five detected objects may be generated for eachof the ten tracked objects based upon match probability information. Theobject tracking module 206 may generate an identification number oranother type of identification symbol for each object that it tracks.The motion modeling module 1002 may use a Kalman filter to predict theposition and velocity of an object.

After the motion modeling module 1002 ranks objects of the current frameby their likelihood of corresponding to tracked objects, the objecttracking module 206 may call on the match classifier 218 to determinewhether an object of the current frame corresponds to a tracked object.For a given tracked object, the object tracking module 206 may call onthe match classifier 218 to first determine whether the highest rankedobject of the current frame matches the tracked object. Moreover, theobject tracking module 206 may use match probability information todetermine an order for the tracked objects. For example, if the motionmodeling module 1002 determines that the probability of a match betweena first tracked object and its highest ranked object is greater than theprobability of a match between a second tracked object and its highestranked object, then the object tracking module 206 may call on the matchclassifier 218 to attempt to match the first tracked object with itshighest ranked object before attempting to match the second trackedobject with its highest ranked object. Although the motion modelingmodule 1002 has been described as being used prior to the matchclassifier 218, the motion modeling module 100 may be used before orafter the match classifier 218. In general the motion modeling module1002 may be used in conjunction with the match classifier 218. Themotion modeling module 1002 may be used to help determine the confidenceof a match or non-match declared by the match classifier 218. Forexample, the match classifier 218 may declare that two objects are amatch, but the motion modeling module 1002 may determine that the twoobjects are too far apart in the field of view to correspond to the sameobject. In that case, the motion modeling module 1002 may be used toincrease the acceptance threshold .tau..sub.a. The match confidencelevel associated with the match classifier 218 may then be analyzed inconjunction with determinations made by the motion modeling module 1002to declare a match or non-match.

FIG. 11 is a flow chart of a tracking process 1100 that may be used bythe object tracking module 206. A first object is captured in a firstimage and detected (step 1102). A first signature of the first object isproduced (step 1104). The first signature may include one or more of thefeatures described above. The first object may be included in the listof tracked objects. A second object is captured in a second image anddetected (step 1106). The object tracking module 206 may use the motionmodeling module 1102 to rank the second object's likelihood ofcorresponding to the first object based upon the first object'sestimated trajectory and velocity (step 1108). A second signature of thesecond object is produced (step 1110). The second signature may includeone or more of the features described above. Step 1110 need not occurafter step 1108 (i.e., the second signature may be produced before themotion modeling module 1102 is used). The first and second signaturesare combined as described above (step 1112). For example, the differentfeatures (e.g., histograms) are combined. The combined signatures aretransmitted to the match classifier 218, and the match classifier 218determines whether the first and second signatures match (step 1114).The match classifier 218 declares a match or a non-match, together witha match confidence level. If the match classifier 218 determines thatthe first and second signatures match, the first signature is updated asdescribed below and other metadata may be generated such as, but notlimited to, motion information (e.g., velocity, trajectory), informationcorresponding to the number of frames in which the first object isdetected, the object's location and size (e.g., the size of a boundingbox surrounding the object), the object's type (e.g., classificationlabel determined by the temporal object classification module 208), aclassification confidence level, and a match confidence level.

As described above, the match classifier 218 may use object signaturesto determine whether objects match (i.e., whether objects are instancesof each other). Different signatures may be generated for each object.For example, a single-frame signature and a cumulative signature may beused by the match classifier 218 during tracking. A single-framesignature (e.g., an instantaneous signature) may represent an object'sappearance in a particular video frame (i.e., a single image of theobject). A single-frame signature may include one or more of thefeatures described above (e.g., histograms) computed based upon thecorresponding frame. A single-frame signature may be generated for everyinstance of an object. The second signature described in the previousparagraph may be a single-frame signature. A cumulative signature may bea weighted average of a sequence of single-frame signatures. Acumulative signature may represent the appearance of an object beingtracked (e.g., an object in the list of tracked objects). Using acumulative signature for a tracked object rather than a single-framesignature may be more robust to noise because a cumulative signature isaveraged over multiple frames. The first signature described in theprevious paragraph may be a cumulative signature.

A cumulative signature may be updated whenever a tracked object ismatched to a new instance by the match classifier 218. For example, thecumulative signature may be updated in real-time as new images of anobject are captured. Moreover, the contribution of each single-framesignature may be reduced (e.g., exponentially reduced) corresponding totime. Thus, the most recent single-frame signatures may contribute themost to a cumulative signature. A cumulative signature may berepresented mathematically as follows:c.sub.object=.rho.c.sub.object+(1−.rho.)i.sub.observation  (13)where c.sub.object represents the cumulative signature,i.sub.observation represents a single-frame signature corresponding toan instance of an object, and .rho. is a discount factor in which0<.rho.<1. When an object is first detected, the cumulative signaturemay be equal to the single-frame signature, and may thereaftercorrespond to equation (13) for subsequent detections.

A third type of signature—an average signature—may also be generated foran object. Like a cumulative signature, an average signature of anobject includes contributions from single-frame signatures of theobject. Unlike the cumulative signature, however, the single-framesignatures contributing to the average signature are not weighteddifferently. In other words, the contributions from single framesignatures are not reduced according to time. Thus, an average signaturemay represent the sum of a set of single-frame signatures divided by thenumber of single-frame signatures of the set. The following equation isa mathematical representation of an average signature a.sub.object of anobject:a object=1n1nki  (14) ##EQU00003##where the object has been tracked for n frames and k.sub.i representsthe single-frame signature for the i.sup.th frame. The average signaturemay be updated in real-time as new single-frame signatures aregenerated.

The category or class of objects may also be taken into considerationduring tracking. In many applications, specific categories or classes(e.g., humans, vehicles, animals) of objects are tracked. The objectcategorization module 216 is operable to categorize an object in eachframe as either one of the classes of interest or as “unknown.” Theobject categorization module 216 is operable to use object classinformation and other information to generate an order in which trackedobjects are matched with objects of a current frame. For example, the“importance” of a tracked object may correspond to the number of framesin which the tracked object was classified as something other than“unknown.” Thus, noisy observations corresponding to moving leaves,water, camera noise and the like may have low importance because thenoisy observations may usually be classified as “unknown.” The objectcategorization module 216 may attempt to match noisy observations last,which may reduce the possibility of erroneously matching an “unknown”observation to a object of interest, such as a human or vehicle. Theobject categorization module 216 may be useful when a large number ofobjects are captured in a current frame and sufficient resources may notbe available to track all of the objects. In that case, the “importance”of tracked objects may be used to determine an order for tracking inwhich the most important objects are tracked first.

FIG. 12 is a hybrid block diagram and flow chart of a method 1200 oftracking an object using class information and the modules involved inthat method. Objects are detected in a current frame and the objectclassification module 210 classifies the objects of the current frame aseither a member of one of the object classes (e.g., human, vehicle) oras “unknown.” Data representing the classified objects 1202 are suppliedto the motion modeling module 902. Data representing a list of trackedobjects are also supplied from the object categorization module 216 tothe motion modeling module 902. The tracked objects are objects thathave been observed in recent frames. The object categorization module216 identifies the “importance” of each tracked object based upon thenumber of frames in which the temporal object classification module 208indicates that the tracked object was classified as a member of a classof interest. For example, a user may specify that humans and/or vehiclesare classes of interest. Thus, the more frames in which a tracked objectis classified by the object classification module 210 as a human orvehicle, the greater the “importance” of the tracked object. The objectcategorization module 216 may sort the list of tracked objects accordingto importance. The data supplied from the object categorization module216 to the motion modeling module 902 indicates the importance of eachof the tracked objects and also includes motion information (e.g.,position, trajectory, velocity) for each of the tracked objects.

The motion modeling module 902 receives the data representing the listof tracked objects and the data representing the classified objects1202. The motion modeling module 902 begins with the most importanttracked object from the list (e.g., a first tracked object) and ranksthe classified objects 1202 based on their likelihood of correspondingto the first tracked object. Signatures of likely matches are combinedwith the signature of the first tracked object, and the combinedsignatures are transmitted to the match classifier 218 to determinewhether a classified object 1202 matches the first tracked object. If aclassified object 1202 matches the first tracked object, motioninformation (e.g., position, trajectory, velocity) and appearanceinformation (e.g., cumulative signature) of the first tracked object areupdated (step 1204). Additionally, when a classified object 1202 matchesthe first tracked object, the class of the classified object istransmitted to the temporal object classification module 208 so that the“importance” of the first tracked object can be updated. If none of theclassified objects 1202 match the first tracked object, information isgenerated to indicate a non-match. The information indicating anon-match may be transmitted to the temporal object classificationmodule 208 and the “importance” of the first tracked object may beupdated.

If a tracked object is not matched for a predetermined amount of time(e.g., the object may have left the scene), the list of tracked objectsmay be updated by removing the tracked object (step 1206). When atracked object is removed from the list, the object's information (e.g.,identification number, signature, class) may be transmitted to themetadata database 112 for storage. Additionally, if a classified object1202 does not match any of the tracked objects, the classified object1202 may be added to the list of tracked objects (step 1206).

The camera system 100 may also be operable to perform inter-cameratracking, or tracking between multiple image capturing devices 102.FIGS. 13A and 13B are hybrid block diagrams and flow charts ofinter-camera tracking processes 1300 and 1301 used by, respectively, atransmitting camera and a receiving camera. A first image capturingdevice 102 (e.g., the transmitting camera) detects an object in itsfield of view and tracks the object for a number of frames. The trackedobject 1302 may have its signature routed to other image capturingdevices 102 (e.g., receiving cameras) (step 1304). The signature that isrouted may be an average signature of the tracked object 1302. Thesignature may be transmitted to all image capturing devices 102 of thecamera system 100, or to certain image capturing devices 102 accordingto a routing table 1306. The routing table 1306 may include informationcorresponding to the probability that an object has appeared or willappear in a receiving camera's field of view within a certain timeperiod after or while the object appears in the transmitting camera'sfield of view. The routing information is generated over time as thetransmitting camera transmits signatures to receiving cameras, and asthe receiving cameras report back to the transmitting camera matchnotification information 1308 indicative of matches and/or non-matches.For example, a receiving camera located far away from the transmittingcamera may rarely, if ever, report a match because an object beingtracked by the transmitting camera may not be captured by the receivingcamera within a predetermined time period (e.g., the object cannot movefast enough between the transmitting camera's field of view and thereceiving camera's field of view). Thus, probability information may beused to selectively transmit signatures to certain cameras so as to notburden other cameras with matching tasks when the probability of theother cameras capturing the object within a predetermined time period islow.

The match notification information 1308 is received by a matchnotification responder 1310 that uses the match notification information1308 to update the probabilities of the routing tables. For example, ifa receiving camera reports a match, the match notification responder1310 may increase the probability in the routing table corresponding tothat receiving camera. The match notification information 1308 may alsoinclude signature information corresponding to the object matched by areceiving camera. The match notification responder may update thesignature of the object when a receiving camera detects a match andstore the updated signature in the metadata database 112. Informationidentifying the receiving camera that matched the object may also bestored in the metadata database 112.

As shown in FIG. 13B, an object signature transmitted from thetransmitting camera to a second image capturing device 102 (e.g.,receiving camera) is stored in a match database 1312 of the receivingcamera. The match database 1312 may be part of the storage system 100.The signature of the object is combined with one or more activesignatures of objects being tracked by the receiving camera. The activesignatures of the objects being tracked may be average signatures andmay be stored in the metadata database 112 of the receiving camera. Thecombined signatures are transmitted to the match classifier 218 of thereceiving camera, and the match classifier 218 attempts to match thesignatures for a predetermined number of frames (step 1314). If thematch classifier 218 determines that the transmitted signature does notmatch one of the active signatures for the predetermined number offrames, the transmitted signature is removed from the match database1312 (step 1316). The routing table of the receiving camera is updated(e.g., the probability of a match corresponding to the transmittingcamera is decreased), and the receiving camera notifies the transmittingcamera of the non-match (step 1318). If the match classifier 218determines that the transmitted signature matches one of the activesignatures, the corresponding active signature may be updated (e.g., thetransmitted signature is averaged with the active signature) and storedin the receiving camera's metadata database 112, and/or the transmittedsignature may be stored in the receiving camera's metadata database 112as a new signature for the object (step 1320). The routing table of thereceiving camera is then updated (e.g., the probability of a matchcorresponding to the transmitting camera is increased) and the receivingcamera notifies the transmitting camera of the match (step 1318).

Inter-camera tracking may be used for automatic camera calibration. Forexample, if the location of an object in the field of view of each ofmultiple image capturing devices 102 is known, and if the approximatesize of the object is known, then the distance between the imagecapturing devices 102 and their relative orientation can be calculated.Following calibration, the location of an object simultaneously trackedby the multiple image capturing devices 102 can be determined inthree-dimensional (3-D) “world” coordinates. Thus, the system cananalyze motion of objects not only in a two-dimensional image plane, butalso in 3-D.

By implementing the object tracking module 206 and the match classifier218, object tracking may be significantly improved over known trackingsystems. Moreover, data generated by the object tracking module 206 maybe supplied to the object detection module 204 and the objectclassification module 210 to improve object detection and objectclassification. Additionally, objects may be accurately tracked inreal-time.

Indexing

The signatures of objects act as index elements for the images or videoclips in which the objects are captured. The index elements enableretrieval of the video clips of the objects when the objects areselected. The signatures may be stored in the metadata database 112. Thesignatures may be compact compared to the images or video clipsassociated with the objects. Moreover, the signatures may enableobject-based searching. Once a signature is generated for an object,tracking and searching processes may be independent of video images ofthe object. Thus, video information need not be uncompressed to performsearches, and the search process may be resilient to uncompressedartifacts and/or poor video quality.

Each object may have more than one signature associated with it. Forexample, an object may have multiple single-frame signatures andcumulative and average signatures comprising the single-framesignatures. An object's average signature may be useful for acting as anindex element for the object.

A single average signature computed over the entire duration in which anobject is tracked, however, may not be suitable to enable object-basedsearching because the appearance of an object may be multi-modal or mayvary over time (e.g., a front view of an object versus a side view,lighting changes, views of different portions of the object). Thus, theobject indexing module 212 is operable to determine when to generate anew average signature for an object. Moreover, the object indexingmodule 212 may be operable to call on the match classifier 218 todetermine whether an object's appearance has changed enough such that anew average signature should be generated for the object.

FIG. 14 is a block diagram showing how the match classifier 218 may beused for indexing. When a new single-frame signature 1402 of an objectis generated from a new instance of an object and the new instance ismatched by the match classifier 218 with a tracked object, the trackedobject's cumulative signature 1404 is updated with respect to thesingle-frame signature 1402. After the cumulative signature 1404 isupdated, the cumulative signature 1404 is combined with the trackedobject's average signature 1406. The combined signatures 1408 issupplied to the match classifier 218. If the match classifier 218determines that the cumulative signature 1404 and the average signature1406 match, the average signature is updated with respect to thesingle-frame signature 1402 (block 1410). If the match classifier 218determines that the cumulative signature 1404 and the average signature1406 do not match, a new average signature is created using thesingle-frame signature 1402 (block 1412). The new average signature maybe stored in the metadata database 112 as a second index element for theobject. If the tracked object has multiple average signatures, thecumulative signature 1404 may be compared to each of the multipleaverage signatures, and a new average signature may be created when thecumulative signature 1404 does not match any of the multiple averagesignatures. As described above, the same match classifier 218 used totrack objects may be used to detect appreciable changes in an object'sappearance, and, thus, a new index element may be generated by using thematch classifier 218.

Searching

The signatures or index elements stored in the metadata database 112 mayfacilitate searching a large database of objects quickly for a specificobject because actual pixel information from video images does not needto be reprocessed. The object search module 214 may use the same matchclassifier 218 used for tracking and indexing to search for a specificobject. The match classifier 218, together with the signatures ofobjects, enable object-based searches in both historical video andreal-time video feeds.

FIG. 15 is a pictorial diagram of a search process 1500 that may beperformed by the camera system 100 by way of example only; the process1500 is operable with any suitable camera system. One of the imagecapturing devices 102 captures an image of a reference object, and theimage of the reference object is presented on the display 114 of theuser interface 104. The reference object may be highlighted for the userby, for example, presenting a box outline around the image of thereference object. The user may use the input device 116 to select thereference object. For example, the input device 116 may include a mouse,and the user may click a pointer on the image of the reference objectusing the mouse. Thus, the image of the reference object may beconsidered a hyper-tag (somewhat similar to a hyper-link in an HTMLpage). When the user selects the reference object, the selection ismapped to the reference object's identification number generated by theobject tracking module 206 of the image capturing device 102 thatcaptured the image of the reference object (step 1502). The referenceobject's identification number is used by the object search module 214to retrieve the reference object's signature from the metadata database112 (step 1504). The signature retrieved may be the reference object'saverage signature, and if the reference object has multiple averagesignatures, the multiple average signatures may be retrieved.

In addition to the average signature, one or more fast signatures of thereference object may be retrieved from the metadata database 112. A fastsignature may be a simple appearance descriptor computed from an averagesignature. The fast signature may be computed by discarding some of thefeatures contained in the average signature by reducing the number ofbins in a histogram, for example, or by computing statistics over thefeatures (e.g., maximum or entropy statistics). The fast signature maybe used to quickly reduce the search space of stored signatures byremoving candidate objects that most likely do not match the firstobject. In addition to the fast signature, other criteria may be used bythe object search module 214 to quickly prune out obvious non-matches.For example, the object search module 214 may use the reference object'sclass to eliminate candidate objects. Moreover, the object search module214 may eliminate candidate objects based on time (e.g., eliminatecandidate objects that were detected prior to a predetermined timeframe) and/or based on one or more rule violations associated with thereference object (e.g., eliminate candidate objects that did not triggersimilar rule violations).

The fast signature(s) of the first object are transmitted to the otherimage capturing devices 102 over the camera network 108. The metadatadatabases 112 of the other image capturing devices 102 are searched forfast signatures of candidate objects that match the fast signature(s) ofthe reference object. The object search modules 214 of the other imagecapturing devices 102 compare the fast signature(s) of the referenceobject with the fast signatures of the candidate objects by computing adistance measure between the signatures. The distance measure may thenbe compared to a predetermined threshold. The threshold may be set suchthat the false negative rate (e.g., rate of discarding matches) is low,but the false positive rate (e.g., rate of accepting non-matches) may behigh. If the object search modules 214 determine that one or more fastsignatures of the candidate objects match the fast signature of thereference object, the object search modules 214 retrieve the averagesignatures of the corresponding candidate objects.

The average signatures of the candidate objects are transmitted to theimage capturing device 102 that captured the image of the referenceobject or to the remote storage/processing unit 106 (step 1510). Thematch classifier 218 of the image capturing device 102 or the matchclassifier 218 of the remote storage/processing unit 106 is used tocompare the average signature(s) of the reference object to the averagesignatures of the candidate objects (step 1512). The match classifier218 compares the average signatures to discard non-matches and to rankmatches based on the match confidence level associated with the matches.Once the matches are ranked, a list including representations of theranked matches may be presented on the display 114 (step 1514).Additionally, video images corresponding to the candidate objects of thematches are retrieved over the camera network 108 from one or more ofthe storage systems 110, or from the remote storage/processing unit 106.The user may select a representation of a candidate object from thelist, and the video images of the selected candidate object may bepresented on the display 114.

The metadata database 112 of the image capturing device 102 thatcaptured the image of the reference object may also be searched toidentify signatures of potential matches, similar to the process 1500described above. For example, the object search module 214 of the imagecapturing device 102 may search the fast signatures first and then callon the match classifier 218 to compare the average signature of thereference object to the average signatures of the potential matches.Moreover, although process 1500 includes matching with the matchclassifiers 218 located at either the image capturing device 102 thatcaptured the image of the reference object or the remotestorage/processing unit 106, a match classifier 218 associated with anyother image capturing device 102 may be used. In other words, matchingmay be performed at one or more image capturing device 102 or at theremote storage/processing unit 106.

The match classifier 218 and the object search module 214 enable a formof searching that may be referred to as “click and search.” A user maybe reviewing historical video from a variety of image capturing devices102. If a particular object appears interesting, the user can search forall objects that look like the particular object. Moreover, video clipsof look-a-like objects may be played for the user. The search may beperformed over all the image capturing devices 102 of the camera system100 or over any user-specified subset. Because the video does not haveto be reprocessed to perform a search, the search may be performedquickly.

The match classifier 218 and the object search module 214 also enable aform of searching that may be referred to as “click and track.” Inaddition to performing searches with respect to historical video,searches may be performed for instances of an object that has justappeared in a live video feed. Much like intra-camera tracking, the userinterface 104 requests (in response to a user selection) that the imagecapturing device 102 that captured an object of interest transmit thesignature of the object the image capturing devices 102 in the network108. If a match is found, the user interface 104 is notified and theimages of the object of interest are presented on the display 114.

While reviewing video data retrieved from an object search, a user canvalidate or invalidate a match. For every match that is validated by auser, a link in the metadata database 112 may be created and thecorresponding signatures may be updated. For invalid matches, signaturesmay be designated as exclusions to matching against the object'ssignature. User validation data can be stored in the metadata databaseand retrieved later to re-train match classifier 218, resulting infurther improvement of search and inter-camera tracking results as wellas intra-camera tracking results. Additionally, user validation may beused to tune search results

Applications

The systems and methods described above may be used to create a numberof applications. Examples of some of these applications are presented inthe following paragraphs.

Compared to known systems that create tracking information only when arule established by a rules engine has been violated, the camera system100 creates rich tracking metadata whether a rule is violated. Thisenables post searching, or searching video images that were capturedbefore a rule is violated. Moreover, because metadata is generated thatincludes object appearance characteristics, an object may be recognizedas a member of a newly created object class even though an image of theobject was captured prior to creation of the new object class.

Known tracking systems have problems with losing track of objects. Forexample, if a person walks behind a park bench, the person's legs mightnot be visible. A known tracking system might consider the person tooshort to be a person. As a result, the person may be tracked until theperson walks behind the park bench. Because the camera system 100 mayuse many different appearance characteristics, the camera system 100 mayavoid losing track of objects when they are still present within thefield of view. Moreover, the match classifier 218 may make tracking moreadaptable to environmental changes and may improve detection acrossdifferent camera types (and from camera to camera in general) comparedto known systems.

Because the match classifier 218 is trainable, new features of objectsmay be selected to improve detection and tracking accuracy. The matchclassifier 218 may learn the features that are most important in thepre-deployment training phase, and may also learn while in the field toimprove tracking accuracy.

By establishing semantic links between video streams and objectsdetected, a video history can be created for a particular object. Forinstance, by selecting a human object, a user may automatically summonvideo clips showing where the person had been detected previously byother cameras. The user may then notice companions of the person inquestion, and may select those companions and view their videohistories. Because metadata corresponding to the object's appearancesignature is linked in the database with video data corresponding to thelocation where it was detected, the image itself may be used as aselectable link for searching the database.

When an object is captured by one image capturing device 102, thesystems and methods described may also be used to notify and engage allother image capturing devices 102, so that they may automatically beginsearching for the same object and create an alert if it is detected. Forexample, an individual person may be followed through a casino monitoredby dozens of cameras with adjacent, overlapping fields of view, by justclicking on an image and instructing the system to track the imageacross all cameras. As soon as an appearance signature of the person isdetected, the system automatically directs live video data from thecorresponding camera to a monitor that allows security personnel tovisually track the person. As the person moves into the field of view ofthe next camera, the video feed is automatically switched so that it isnot necessary for security personnel to switch back and forth betweencameras to continue tracking the person's path.

The camera system 100 may automatically recognize pedestrians leaving aparked car, and can compare them later to people entering the car. If aperson entering a car is not from the original group who arrived in thecar, security personnel may be alerted. Video clips are automaticallysent with the alert, so that it is easy to review and quickly determinewhether there is a problem. A security guard may then opt to eitherinform the car owner (if the car is registered by license plate number)or summon police.

With the ability to track objects, fraud can be detected by followingthe path of individual shoppers through a retail store. With thiscapability the system disclosed may be taught to verify whether shoppersexiting a store with merchandise have visited a checkout counter andpurchased the merchandise before leaving the store. The system may alsodetermine whether customers at a return counter came straight from theircars, or whether they first passed through the store, selected productsfrom the store, and attempted to return them as though they hadpurchased the products. The return counter attendant can be shown a clipof the customer entering the store and their path through the store, asverification.

When combined with object matching, performance of facial recognitionand vehicle license plate recognition systems can be improved. Objectmatching may provide good tracking capability by overall appearance andmovement and can track people in any orientation, not just when they arealigned to face the camera.

The camera system 100 may also be configured to automatically recognizeabnormal activity by monitoring vehicle and pedestrian patterns overdays or weeks. High risk facilities concerned about potential terroristactivities may benefit from this feature. If the camera system 100repeatedly recognizes a vehicle parked for an extended period of timewith no one exiting the vehicle, an alert of suspicious activity may begenerated to summon security personnel to review the video data.Alternatively, object matching combined with a facial recognition systemmay recognize individuals who walk around a facility repeatedly. Suchaction may catch a criminal casing a facility. In another example, ifuniformed employees regularly visit a special lunch room or inventoryroom, for example, the camera system 100 can recognize this pattern.Later, if someone not wearing such a uniform enters one of these rooms,the camera system 100 may respond with an automatic alert.

The camera system 100 may be used to measure how many people arestanding in a line at, for example, a cash register and to generate analert corresponding to a rule of a rules engine. Additionally, thecamera system 100 may be used to measure how long each person has beenstanding in the line because the system 100 can track each individual.If a person has been standing in the line too long, a rule of the rulesengine may be violated and an alert may be generated to summonassistance (e.g., an alert to open a new cash register line). Moreover,statistics may be generated for managers to help measure customerservice performance.

The camera system 100 may be used to help prevent child abductions atretail businesses for example. The camera system 100 may recognizechildren when they enter a store and an adult accompanying the child.The camera system 100 may also recognize an adult the child stays closeto while walking through the store. If the child leaves the storewithout the adult accompanying the child, an alert may be created. Thecamera system 100 may also identify the location of the adult so thatshe/he can be notified.

The camera system 100 may be used to locate lost children at retailbusinesses. For example, if a parent loses its child and notifies asecurity guard, the security guard can find an image of the parent,click on the image to search through other video images and identify animage when the parent entered the store with the child. The securityguard may then click on an image of the child and search the imagecapturing device 102 that most recently captured an image of the child.This quickly locates the child's location in the store.

The camera system 100 may be use to prevent a problem called “sweetheartfraud,” which happens when a checkout person intentionally swipes thebar code of a low cost item while acting as though they are swiping thebar code of a more expensive item. The camera system 100 may beconnected to the point-of-sale (POS) data outputs of a cash register andmay compare the appearance of products with the bar code information.Over a period of time, the camera system 100 may automatically create inthe metadata database 112 appearance signatures for different products.After the camera system 100 has reached a level of accuracy for productrecognition, which the camera system 100 can determine itself, it canautomatically begin creating alerts whenever the bar code informationdoes not match the image being scanned.

The camera system 100 may automatically tag video for higher prioritybased on its similarity to footage in which a critical event isdetected. For example, if a laptop is stolen from a work area, thecamera system 100 may be able to automatically recognize the laptop fromhistorical video and may recognize the person closest to the laptop whenit went missing. The camera system 100 may then automatically designateall video images of the suspicious person to be of higher priority forstorage, retention, or retrieval and review by security personnel.

CONCLUSION

The methods and systems illustrated and described herein can exist in avariety of forms both active and inactive. For example, they can existpartly or wholly as one or more software programs comprised of programinstructions in source code, object code, executable code or otherformats. Any of the above can be embodied in compressed or uncompressedform on a computer-readable medium, which include storage devices.Exemplary computer-readable storage devices include conventionalcomputer system RAM (random access memory), ROM (read only memory),EPROM (erasable, programmable ROM), EEPROM (electrically erasable,programmable ROM), flash memory and magnetic or optical disks or tapes.

The terms and descriptions used above are set forth by way ofillustration only and are not meant as limitations. For example, thematch classifier 218 may be part of—and the classification methods maybe performed at—a remote processing unit, such as the remotestorage/processing unit 106 (FIG. 1), a computer associated with theuser interface 104, another node in the camera network 108, or anotherserver, such as one at a central location or on another network. Thoseskilled in the art will recognize that many variations, enhancements andmodifications of the concepts described herein are possible withoutdeparting from the underlying principles of the invention. The scope ofthe invention should therefore be determined only by the followingclaims and their equivalents.

What is claimed:
 1. A system comprising: a display; an input device; aprocessor communicatively coupled to the display and the input device;and a memory communicatively coupled to the processor and having storedthereon computer program code that is executable by the processor,wherein the computer program code, when executed by the processor,causes the processor to: receive image data representing a first objectin an image of a field of view of a camera system, the first objecthaving a first signature representing features of the first object;display an image of the first object on the display; accept input fromthe input device indicative of a selection of the first object; makedeterminations as to whether the first signature matches one or moresignatures of a group that correspond to a set of objects captured inimages of the field of view; and send a notification indicating a matchbetween the first signature and a second signature of a second object ofthe set.
 2. The system of claim 1, in which the notification includesdisplaying an image of the second object on the display.
 3. The systemof claim 1, in which the camera system comprises first and second imagecapturing devices connected to one another and the display, the firstand second image capturing devices having, respectively, a first fieldof view and a second field of view, the field of view of the camerasystem corresponding to the first field of view and the second field ofview in combination, wherein the image of the first object is capturedwith the first image capturing device and a second image of the secondobject is captured with the second image capturing device.
 4. The systemof claim 1, in which the first object includes a fast signaturerepresenting a simplified version of the first signature, and theobjects of the set include fast signatures representing simplifiedversions of the group of signatures, wherein the computer program code,when executed by the processor, further compares the fast signature ofthe first object with the fast signatures of the objects of the setprior to making determinations as to whether the signatures of the groupmatch the first signature.
 5. The system of claim 1, wherein thecomputer program code, when executed by the processor, further retrievesthe group of signatures from a database.
 6. The system of claim 1,wherein the computer program code, when executed by the processor,further sends the notification when the second signature matches thefirst signature and when a selected rules violation is associated withthe second object.
 7. The system of claim 1, in which a rule violationis associated with each of the objects of the set.
 8. A non-transitorycomputer readable medium having stored thereon computer program codethat is executable by a processor and that, when executed by theprocessor, causes the processor to: receive image data representing afirst object in an image of a field of view of a camera system, thefirst object having a first signature representing features of the firstobject; display an image of the first object on the display; acceptinput from the input device indicative of a selection of the firstobject; make determinations as to whether the first signature matchesone or more signatures of a group that correspond to a set of objectscaptured in images of the field of view; and send a notificationindicating a match between the first signature and a second signature ofa second object of the set.
 9. The non-transitory computer readablemedium of claim 8, in which the notification includes displaying animage of the second object on the display.
 10. The non-transitorycomputer readable medium of claim 8, in which the camera systemcomprises first and second image capturing devices connected to oneanother and the display, the first and second image capturing deviceshaving, respectively, a first field of view and a second field of view,the field of view of the camera system corresponding to the first fieldof view and the second field of view in combination, wherein the imageof the first object is captured with the first image capturing deviceand a second image of the second object is captured with the secondimage capturing device.
 11. The non-transitory computer readable mediumof claim 8, in which the first object includes a fast signaturerepresenting a simplified version of the first signature, and theobjects of the set include fast signatures representing simplifiedversions of the group of signatures, wherein the computer program code,when executed by the processor, further compares the fast signature ofthe first object with the fast signatures of the objects of the setprior to making determinations as to whether the signatures of the groupmatch the first signature.
 12. The non-transitory computer readablemedium of claim 8, wherein the computer program code, when executed bythe processor, further retrieves the group of signatures from adatabase.
 13. The non-transitory computer readable medium of claim 8,wherein the computer program code, when executed by the processor,further sends the notification when the second signature matches thefirst signature and when a selected rules violation is associated withthe second object.
 14. The non-transitory computer readable medium ofclaim 8, in which a rule violation is associated with each of theobjects of the set.